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ABSTRACT 

NATO  aspires  to  be  the  final  arbitrating  body  in  developing  agreed  definitions  between  its  member  nations. 
The  reality  is  that  many  NATO  definitions  in  use  are  what  can  be  agreed  upon  between  the  disparate  parties, 
rather  than  being  rigorously  derived.  Frequently,  they  are  ambiguous  and  vague  and  consequently  they  offer 
little  to  direct  or  inform  analysis  work.  An  ambiguous  and  vague  definition  can  be  better  than  nothing,  but 
NATO  should  aspire  to  improve  the  manner  in  which  definitions  are  derived,  since  they  form  the  fundamental 
building-blocks  on  which  conceptual  and  capability  efforts  are  built. 

The  paper  dwells  on  a  recent  case  study  of  a  requirements  analysis  undertaken  for  NATO  Expeditionary 
Operations.  Specifically,  the  presentation  covers  the  problems  of  the  definition  of  Expeditionary  Operations  in 
use  within  NATO  and  the  methods  employed  by  ACT  analysts  to  develop  an  improved  definition. 


INTRODUCTION 

NATO  aspires  to  be  the  final  arbitrating  body  in  developing  agreed  definitions  between  its  member  nations. 
The  reality  is  that  many  of  the  definitions  in  use  within  NATO  are  what  could  be  agreed  upon  between  various 
parties  holding  different  views,  rather  than  having  being  rigorously  derived.  Frequently,  NATO  definitions  are 
ambiguous  and  vague  and  offer  little  to  direct  or  inform  supporting  work.  In  some  cases,  an  agreeable  but 
ambiguous  definition  may  be  better  than  nothing,  but  NATO  should  aspire  to  improve  the  rigour  with  which 
definitions  are  derived,  since  they  form  the  fundamental  building-blocks  on  which  conceptual  and  capability 
efforts  are  built. 

This  paper  presents  an  analysis  method  employed  by  a  Specialist  Team  on  Characteristics  of  Operations  for 
Future  Expeditionary  Forces  (SAS-075).  This  paper  discusses  the  method  selected,  General  Morphological 
Analysis,  its  benefits  and  limitations,  and  how  the  team  overcame  these  limitations  by  combining  it  in  a  multi¬ 
method  approach. 


BACKGROUND 

Rittel  &  Webber  wrote  about  two  types  of  problems:  “Tame”  and  “Wicked”1.  Tame  problems  were  well 
defined,  easily  quantifiable,  and  had  definite  solutions  that  could  be  measured  and  compared  to  similar 
problems  that  had  been  solved  successfully  before.  Conversely,  Wicked  problems  were  unique,  hard  to  define 


1  Rittel  &  Webber,  “Dilemmas  in  a  General  Theory  of  Planning”,  1973. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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or  bound,  they  had  no  stopping  rules,  no  obvious  solutions  nor  any  immediate  test  of  the  quality  of  the 
solution. 

There  are  different  strategies  for  approaching  Wicked  problems2: 

•  Authoritative  (a  small  group  is  formed  to  tackle  the  problem); 

•  Competitive  (the  problem  is  opened  up  to  competing  groups)  and; 

•  Collaborative  (a  wide  range  of  views  taken  into  account). 


METHOD 

General  Morphological  Analysis 

Developed  by  Zwicky,  General  Morphological  Analysis  breaks  down  a  problem  into  its  key  dimensions, 
restructures  it  and  provides  a  framework  in  which  people  can  consider  various  solutions.  It  offers  a  method  of 
presenting  a  multi-dimensional  problem  in  a  two-dimensional  space  in  order  to  ease  understanding.  General 
Morphological  Analysis  may  be  considered  to  be  an  authoritative  approach;  since  it  brings  together  a  small 
group  of  experts. 

The  first  step  in  General  Morphological  Analysis  is  to  create  a  Morphological  Table;  an  example  of  a 
Morphological  Table  is  shown  in  Figure  1.  This  table  was  developed  by  the  SAS-075  team  to  address  the 
specific  question: 

“ What  is  NATO’s  definition  of  an  Expeditionary  Operation ?” 

Figure  1  shows  how  the  Morphological  table  is  used  to  break  down  a  problem  into  its  key  components.  The 
first  row  of  Figure  1  (in  grey)  shows  the  dimensions3  of  the  problem;  under  each  dimension  are  scalars4, 
which  give  different  value  options  for  each  dimension.  In  this  instance,  the  table  is  intended  to  represent  the 
whole  intellectual  space,  of  which  the  solution  (the  answer  to  the  question  above)  is  some  portion  that 
describes  the  sub-set  of  operations  that  are  considered  to  be  Expeditionary. 


2 

Roberts,  N.C.  “Wicked  Problems  and  Network  Approaches  to  Resolution.,”  The  International  Public  Management  Review.,  Vol. 
1,  2000. 

3 

Called  ‘parameters’  in  General  Morphological  Analysis  terminology. 

4  Called  ‘parameter  value  range’  in  General  Morphological  Analysis  terminology. 
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Figure  1 :  NATO  Operations  Morphological  Table 


The  General  Morphological  Analysis  typically  takes  place  with  a  small  group  of  subject  matter  experts 
through  a  series  of  workshops,  and  there  are  many  examples  of  it  being  used  successfully  in  practice5. 

Limitations  of  General  Morphological  Analysis 

General  Morphological  Analysis  is  a  very  useful  method  for  problem  structuring  and  solution  consideration; 
however,  it  works  best  when  the  Morphological  table  can  be  limited  to  fewer  than  seven  dimensions. 
NATO’s  problems  are  often  very  complex  and  in  this  instance  a  larger  table  was  needed  to  describe  the 
problem  fully.  Using  a  classic  Morphological  Analysis  approach  with  a  large  table,  the  data  set  quickly 
becomes  unmanageable  for  a  group  to  consider,  for  example,  the  table  in  Figure  1,  the  data  set  consists  of 
42,336  potential  operations6. 

NATO  also  requires  consultation  with  at  least  twenty-eight  nations,  plus  academia  and  other  agencies,  spread 
out  over  a  wide  geographic  area.  Gathering  everyone’s  opinions  can  be  impractical  for  a  single  or  even  a 
series  of  small  workshops.  A  way  of  remotely  capturing  opinions  on  an  individual  basis  plus  a  way  of 
consolidating  those  opinions  is  highly  desirable,  as  more  experts  can  be  reached,  as  well  as  reducing  costs  and 
travel  time  for  the  analysts.  General  Morphological  Analysis  does  not  offer  this. 

Data  Capture 

A  tool  has  been  created  to  capture  opinions  based  on  any  Morphological  Table  and  this  creates  a  resulting  data 
set  for  analysis.  This  tool  can  be  sent  via  email  to  as  many  people  as  desired,  or  it  can  be  used  by  the  analyst 
to  capture  opinions  from  individuals  or  small  groups  through  interactive  discussions. 


5  See  www.swemorph.com  for  examples. 

6  The  number  of  potential  combinations  can  be  calculated  by  multiplying  the  number  of  scalars  in  each  dimension  e.g.  Speed  of 
Response  (3)  x  Duration  (2)  x  Distance  (3)  x  Scale  (4)  x  Mission  (7)  x  Human  Environment  (3)  x  Physical  Environment  (7)  x 
Infrastructure  (3)  x  Threat  (2)  =  3x2x3x4x7x3x7x3x2  =  42336. 
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There  are  advantages  and  disadvantages  to  both  methods.  Sending  it  to  individuals  allows  a  large  group  to  be 
targeted  with  minimal  effort  from  an  analyst,  but  with  the  usual  issues  with  response  rate.  Employing  the  tool 
to  collect  data  in  individual  or  small  group  discussions  takes  more  analyst  time;  however  the  confidence  in  the 
quality  of  the  data  increases,  for  example,  workshop  participants  can  be  encouraged  to  consider  about  each 
answer  carefully,  with  the  discussions  translated  by  analysts  who  are  experts  in  the  tool. 

The  tool  currently  uses  MS  Excel  software  and  Visual  Basic  for  Applications.  Excel  was  used  because  it 
provided  all  of  the  necessary  functionality  and  a  graphical  user  interface,  as  well  as  being  widely  understood 
and  accessible  by  other  users,  which  is  important  for  remote  data  collection. 

The  tool  is  in  two  parts.  The  first  requires  the  respondent  to  consider  each  scalar  individually  and  remove  any 
irrelevant  scalars  from  the  list.  The  tool  then  does  a  pair-wise  comparison  between  all  remaining  scalars  and 
asks  the  expert  to  identify  valid  and  erroneous  combinations. 

The  tool  is  interactive  and  dynamic  -  it  takes  user  input  and  then  dynamically  changes  the  questions 
depending  on  the  earlier  answers.  This  reduces  quickly  the  number  of  questions  a  participant  has  to  answer, 
and  therefore  large  Morphological  tables  can  be  examined.  For  example,  using  the  tool  to  examine  the  table  in 
Figure  1  results  in  over  500  potential  questions,  but  this  is  quickly  reduced  and  a  typical  time  to  respond  using 
the  data  capture  tool  is  around  30  minutes. 

The  tool  displays  the  results  of  a  single  response  in  a  Cross-Consistency  Matrix.  Currently  the  tool  requires 
responses  in  the  form  of  “Yes”,  “Maybe”  or  “No”  identifying  the  validity  of  each  pair  of  scalars,  but  the 
responses  can  be  in  any  form  defined  by  the  analyst. 

An  example  of  a  Cross-Consistency  Matrix  produced  by  the  tool  is  shown  in  Figure  2. 
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Figure  2:  Cross  Consistency  Matrix  with  example  data 


Data  from  each  individual’s  Cross-Consistency  Matrix  are  transferred  into  an  analysis  tool  that  combines  the 
results  and  extracts  a  full  list  of  valid  combinations  of  scalars,  or  in  the  example  above,  valid  NATO 
Expeditionary  Operations. 

The  next  step  in  the  General  Morphological  Analysis  process  would  be  to  eliminate  non-valid  combinations  of 
scalars.  By  eliminating  these,  the  number  of  combinations  to  consider  reduces  considerably.  Non-valid 
combinations  can  be  classified  as: 

Logical  Contradictions : 

•  A  logical  contradiction  between  two  or  more  statements; 

•  e.g.  An  operation  in  the  Antarctic  that  is  within  NATO. 

Empirical  Inconsistencies: 

•  A  practical  incompatibility  or  discrepancy  between  two  or  more  conditions  or  statements  about  the 
observed  world; 

•  e.g.  An  urban  operation  at  Global  scale  -  this  is  not  impossible,  just  extremely  unlikely. 
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Normative  Constraints : 

•  An  incompatibility  or  discrepancy  between  two  or  more  conditions  based  on  social  norms,  ethics  and 
standards; 

•  e.g.  A  non-permissive  humanitarian  operation. 

The  final  step  in  General  Morphological  Analysis  is  to  discuss  remaining  valid  combinations  as  potential 
solutions  to  the  problem.  In  the  example  given  in  Figure  1,  a  valid  combination  would  be  a  single  scalar 
selected  from  each  of  the  nine  dimensions  that  together  to  describe  a  potential  Expeditionary  Operation. 

Data  can  be  consolidated  by  listing  all  potential  combinations  and  counting  the  number  of  times  each 
combination  was  identified  as  valid  by  the  respondents.  Depending  on  the  number  of  respondents,  a 
minimum  number  of  votes  for  each  combination  will  need  to  be  determined  by  the  analyst;  however  this  can 
be  done  by  looking  intelligently  at  the  data  and  conducting  sensitivity  analysis  to  ensure  the  chosen  cut  off 
point  does  not  skew  the  results. 

Classification  Trees 

The  next  step  in  the  process  is  to  employ  Classification  Trees  to  analyse  the  consolidated  data.  In  this  case 
SPSS  software7  was  used  to  process  data;  however  other  appropriate  statistical  packages  could  be  used. 

A  Classification  Tree  is  a  data  mining  technique  used  to  predict  the  membership  of  a  dependant  variable  using 
predictor  variables.  The  dependant  variable  is  the  response  to  the  question  “ Is  this  combination  of  scalars  a 
valid  NATO  Expeditionary  Operation?”  answers  to  which  were  captured  in  the  data  capture  tool.  The 
predictor  variables  are  the  dimensions  and  scalars  in  the  Morphological  Table  (Figure  1).  For  any  combination 
of  scalars,  the  Classification  Trees  predict  whether  the  combination  is  likely  to  be  valid,  based  on  the 
collective  responses.  It  shows  the  degree  of  consensus  within  the  data  set.  The  data  set  can  also  be  broken 
down  by  type  of  respondent,  for  example,  civilian  versus  military  respondents. 

A  Classification  Tree  determines  the  relative  importance  of  dimensions  and  scalars  when  predicting 
membership  of  the  dependent  variable.  In  the  example,  Figure  3,  the  first  level  of  the  tree  shows  that  7,510 
out  of  42,336  operations  were  determined  by  respondents  to  be  valid  Expeditionary  Operations. 

Figure  3  also  shows  that  respondents  considered  that  a  majority  of  valid  Expeditionary  Operations  were 
conducted  outside  of  NATO’s  boundaries;  this  was  the  most  significant  predictor  value.  The  next  most 
significant  predictor  was  operation  scale;  with  an  overwhelming  majority  of  Expeditionary  Operations 
considered  to  be  smaller  than  Global. 


7 


See  www.spss.com  for  more  details 


8-6 


RTO-MP-SAS-081 


Getting  Off  to  a  Good  Start: 
Improving  Definitions  Using  Morphological  Analysis 


Figure  3:  Example  of  a  Classification  tree  down  to  3  levels 


Figure  4  shows  the  Classification  Tree  data  in  the  form  of  a  scale  Venn  diagram.  The  outer  circle  represents 
the  whole  data  set,  with  the  Red  and  Blue  areas  representing  all  valid  combinations,  distinguished  by  the 
predictor  variable  Out  of  or  Within  NATO. 


Figure  4:  Venn  diagram  resulting  from  classification  tree  analysis 


Predictive  Rule  Sets 

By  analysing  the  classification  tree,  the  analyst  can  build  rule  sets  that  model  different  problem  definitions  or 
different  solutions  to  the  original  question.  Figure  5  shows  that  four  rules  are  necessary  to  describe  a  large 
proportion  of  Expeditionary  Operations. 
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Figure  5:  Scale  Venn  diagram  showing  Expeditionary  Operations  Rule  Sets 


Rule  Set  Accuracy 

The  rules  depicted  in  Figure  5  can  be  measured  for  accuracy.  This  can  be  summarised  using  a  confusion 
matrix,  shown  in  Figure  6.  This  can  show  the  number  of  correct  predictions  made  by  the  rules,  and  the 
number  of  Type  1  &  2  errors  (False  positives  &  negatives). 
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Figure  6:  Confusion  Matrix 


A  number  of  metrics  can  be  calculated  from  the  confusion  matrix: 

•  Sensitivity  or  Recall:  A/(A+B):  A  measure  of  the  mis-classifications  i.e.  how  many  combinations  are 
classified  as  valid  by  the  rule  set  that  are  actually  not  valid; 

•  Precision:  A  /  (A+C):  This  is  the  percentage  of  all  valid  combinations  that  are  covered  by  the  rule; 
and 

•  Model  Accuracy:  (A+D)  /  (A+B+C+D):  An  overall  measure  of  model  accuracy. 
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Comparison  of  Opinions 

Individual  or  group  opinions  can  be  compared  to  each  other  using  the  accuracy  measures.  In  Figure  7, 
existing  definitions  for  ‘Expeditionary’  are  compared  against  the  rules  generated  through  the  data  capture 
process  using  measures  of  their  precision  (coverage  of  the  Expeditionary  space)  and  sensitivity  (ability  to 
identify  correctly  something  as  Expeditionary).  Better  definitions  score  higher  on  both  axes;  however 
different  strategies  may  trade  one  off  against  another  depending  on  whether  it  is  more  important  to  cover  all 
opinions  (“the  NATO  way”)  or  to  have  very  few  incorrect  predictions.  In  addition  to  collating  individual  or 
group  opinions,  the  method  allowed  for  key  documents  to  be  included  in  the  visualisation  -  with  a  survey 
completed  ‘on  behalf  of  the  document.  The  views  solicited  via  the  Morphological  Analysis  are  in  Green,  Key 
NATO  documents  are  in  Blue,  and  a  USA  Marine  Corps  document  in  Yellow. 

Figure  7  shows  that  the  points  connected  by  the  Red  line  are  on  a  boundary;  points  on  the  line  are  the  best 
options  to  select  for  different  prioritisations  of  Sensitivity  and  Precision,  other  points  (the  Concept  Vision  and 
Framework  and  the  EO  Vision  for  Shaping  Future  Capabilities)  are  dominated.  We  can  improve  the 
sensitivity  of  a  definition  from  the  current  definition  by  selecting  any  of  the  generated  options  or  the  US 
Marine  Corps  definition,  but  Precision  is  lost.  Overall,  a  small  loss  of  Precision  may  be  considered  a 
reasonable  trade  for  the  large  increase  in  Sensitivity. 


Measuring  Accuracy  of  Different  Definitions 


Figure  7:  Comparison  of  different  Existing  Definitions  against  the  New  Rule  Set 


SUMMARY 

Figure  8  summarises  the  basic  analytical  process,  and  shows  how  the  process  has  been  extended  from  the 
classic  General  Morphological  Analysis  approach.  Note  the  cyclical  nature  of  creating  a  rule  set,  testing  for 
accuracy  and  modifying  the  rule  set  to  improve  accuracy  as  necessary. 
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Figure  8:  Summary  of  Analytical  Process 


S AS -075  has  developed  a  generic  method  that  enables  disparate  views  on  a  complex  problem  to  be  elicited 
and  consolidated  in  an  analytically  rigorous  manner.  Although  targeted  at  a  specific  problem,  this  method  has 
the  potential  for  wide  utility  within  NATO  and  national  Capability  Development  domains. 
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